System and methods for providing automatic classification of media entities according to tempo

ABSTRACT

In connection with a classification system for classifying media entities that merges perceptual classification techniques and digital signal processing classification techniques for improved classification of media entities, a system and methods are provided for automatically classifying and characterizing tempo properties of media entities. Such a system and methods may be useful for the indexing of a database or other storage collection of media entities, such as media entities that are audio files, or have portions that are audio files. The methods also help to determine media entities that have similar, or dissimilar as a request may indicate, tempo(s) by utilizing classification chain techniques that test distances between media entities in terms of their properties. For example, a neighborhood of songs may be determined within which each song has similar tempo characteristics.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.09/905,345, filed Jul. 13, 2001, which claims benefit of U.S.Provisional Application Ser. No. 60/218,289, filed Jul. 14, 2000 andrelates to U.S. patent application Ser. No. 09/900,230 filed Jul. 6,2001 (Attorney Docket No. MSFT-0577); Ser. No. 09/899,490 filed Jul. 5,2001, now U.S. Pat. No. 6,545,209 issued Apr. 8, 2003 (Attorney DocketNo.: MSFT-0578); Ser. No. 09/934,071 filed Aug. 20, 2001 (AttorneyDocket No.: MSFT-0579); Ser. No. 09/900,059 filed Jul. 6, 2001 (AttorneyDocket No.: MSFT-0580); Ser. No. 09/934,774 filed Aug. 21, 2001(Attorney Docket No.: MSFT-0581); Ser. No. 09/935,349 filed Aug. 21,2001 (Attorney Docket No.: MSFT-0582); Ser. No. 09/904,465 filed Jul.13, 2001 (Attorney Docket No.: MSFT-0584); Ser. No. 09/905,011 filedJul. 13, 2001 (Attorney Docket No.: MSFT-0585); Ser. No. 09/942,509filed Aug. 29, 2001 (Attorney Docket No.: MSFT-0586); and Ser. No.09/928,004 filed Aug. 10, 2001 (Attorney Docket No.: MSFT-0587).

FIELD OF THE INVENTION

The present invention relates to a system and methods for providingautomatic classification of media entities according to tempoproperties. More particularly, the present invention relates to a systemand methods for automatically classifying media entities according toperceptual tempo properties and tempo properties determined by digitalsignal processing techniques.

BACKGROUND OF THE INVENTION

Classifying information that has subjectively perceived attributes orcharacteristics is difficult. When the information is one or moremusical compositions, classification is complicated by the widelyvarying subjective perceptions of the musical compositions by differentlisteners. One listener may perceive a particular musical composition as“hauntingly beautiful” whereas another may perceive the same compositionas “annoyingly twangy.”

In the classical music context, musicologists have developed names forvarious attributes of musical compositions. Terms such as adagio,fortissimo, or allegro broadly describe the strength with whichinstruments in an orchestra should be played to properly render amusical composition from sheet music. In the popular music context,there is less agreement upon proper terminology. Composers indicate howto render their musical compositions with annotations such as brightly,softly, etc., but there is no consistent, concise, agreed-upon systemfor such annotations.

As a result of rapid movement of musical recordings from sheet music topre-recorded analog media to digital storage and retrieval technologies,this problem has become acute. In particular, as large libraries ofdigital musical recordings have become available through global computernetworks, a need has developed to classify individual musicalcompositions in a quantitative manner based on highly subjectivefeatures, in order to facilitate rapid search and retrieval of largecollections of compositions.

Musical compositions and other information are now widely available forsampling and purchase over global computer networks through onlinemerchants such as AMAZON.COM®, BARNESANDNOBLE.COM®, CDNOW.COM®, etc. Aprospective consumer can use a computer system equipped with a standardWeb browser to contact an online merchant, browse an online catalog ofpre-recorded music, select a song or collection of songs (“album”), andpurchase the song or album for shipment direct to the consumer. In thiscontext, online merchants and others desire to assist the consumer inmaking a purchase selection and desire to suggest possible selectionsfor purchase. However, current classification systems and search andretrieval systems are inadequate for these tasks.

A variety of inadequate classification and search approaches are nowused. In one approach, a consumer selects a musical composition forlistening or for purchase based on past positive experience with thesame artist or with similar music. This approach has a significantdisadvantage in that it involves guessing because the consumer has nofamiliarity with the musical composition that is selected.

In another approach, a merchant classifies musical compositions intobroad categories or genres. The disadvantage of this approach is thattypically the genres are too broad. For example, a wide variety ofqualitatively different albums and songs may be classified in the genreof “Popular Music” or “Rock and Roll.”

In still another approach, an online merchant presents a search page toa client associated with the consumer. The merchant receives selectioncriteria from the client for use in searching the merchant's catalog ordatabase of available music. Normally the selection criteria are limitedto song name, album title, or artist name. The merchant searches thedatabase based on the selection criteria and returns a list of matchingresults to the client. The client selects one item in the list andreceives further, detailed information about that item. The merchantalso creates and returns one or more critics' reviews, customer reviews,or past purchase information associated with the item.

For example, the merchant may present a review by a music critic of amagazine that critiques the album selected by the client. The merchantmay also present informal reviews of the album that have been previouslyentered into the system by other consumers. Further, the merchant maypresent suggestions of related music based on prior purchases of others.For example, in the approach of AMAZON.COM®, when a client requestsdetailed information about a particular album or song, the systemdisplays information stating, “People who bought this album also bought. . . ” followed by a list of other albums or songs. The list of otheralbums or songs is derived from actual purchase experience of thesystem. This is called “collaborative filtering.”

However, this approach has a significant disadvantage, namely that thesuggested albums or songs are based on extrinsic similarity as indicatedby purchase decisions of others, rather than based upon objectivesimilarity of intrinsic attributes of a requested album or song and thesuggested albums or songs. A decision by another consumer to purchasetwo albums at the same time does not indicate that the two albums areobjectively similar or even that the consumer liked both. For example,the consumer might have bought one for the consumer and the second for athird party having greatly differing subjective taste than the consumer.As a result, some pundits have termed the prior approach as the “greaterfools” approach because it relies on the judgment of others.

Another disadvantage of collaborative filtering is that output data isnormally available only for complete albums and not for individualsongs. Thus, a first album that the consumer likes may be broadlysimilar to second album, but the second album may contain individualsongs that are strikingly dissimilar from the first album, and theconsumer has no way to detect or act on such dissimilarity.

Still another disadvantage of collaborative filtering is that itrequires a large mass of historical data in order to provide usefulsearch results. The search results indicating what others bought areonly useful after a large number of transactions, so that meaningfulpatterns and meaningful similarity emerge. Moreover, early transactionstend to over-influence later buyers, and popular titles tend toself-perpetuate.

In a related approach, the merchant may present information describing asong or an album that is prepared and distributed by the recordingartist, a record label, or other entities that are commerciallyassociated with the recording. A disadvantage of this information isthat it may be biased, it may deliberately mischaracterize the recordingin the hope of increasing its sales, and it is normally based oninconsistent terms and meanings.

In still another approach, digital signal processing (DSP) analysis isused to try to match characteristics from song to song, but DSP analysisalone has proven to be insufficient for classification purposes.

U.S. Pat. No. 5,918,223, assigned to Muscle Fish, a corporation ofBerkeley, Calif. (hereinafter the Muscle Fish Patent), describes onesuch DSP analysis technique. The Muscle Fish Patent describes a systemhaving two basic components, typically implemented as software runningon a digital computer. The two components are the analysis of sounds(digital audio data), and the retrieval of these sounds based uponstatistical or frame-by-frame comparisons of the analysis results. Inthat system, the process first measures a variety of acoustical featuresof each sound file and the choice of which acoustical features tomeasure is critical to the success of the process. Loudness, bass,pitch, brightness, bandwidth, and Mel-frequency cepstral coefficients(MFCCs) at periodic intervals (referred to as “frames”) over the lengthof the sound file are measured. The per-frame values are optionallystored, for applications that require that level of detail. Next, theper-frame first derivative of each of these features is computed.Specific statistical measurements, namely, the mean and standarddeviation, of each of these features, including the first derivatives,are computed to describe their variation over time. This set ofstatistical measurements is represented as an N-vector (a vector with Nelements), referred to as the rhythm feature vector for music.

Once the feature vector of the sound file has been stored in a databasewith a corresponding link to the original data file, the user can querythe database in order to access the corresponding sound files. Thedatabase system must be able to measure the distance in N-space betweentwo N-vectors.

Users are allowed to search the sound file database by four specificmethods, enumerated below. The result of these searches is a list ofsound files rank-ordered by distance from the specified N-vector, whichcorresponds to sound files that are most similar to the specifiedN-vector or average N-vector of a user grouping of songs.

-   -   1) Simile: The user may ask for sounds that are similar to an        example sound file, or a list of example sound files.    -   2) Acoustical/perceptual features: The user may ask for sounds        in terms of commonly understood physical characteristics, such        as brightness, pitch and loudness.    -   3) Subjective features: The user may ask for sounds using        individually defined classes. For example, a user might be        looking for a sound that is both “shimmering” and “rough,” where        the classes “shimmering” and “rough” have been previously        defined by a grouping. The user can thus create classes of        sounds (e.g. “bird sounds”, “rock music”, etc.) by specifying a        set of sound files that belong to this class. The average        N-vector of these sound files will represent this sound class in        N-space for purposes of searching. However, this requires ex        post facto user grouping of songs that the user thinks are        similar.    -   4) Onomatopoeia: producing a sound similar in some quality to        the sound you are looking for. For example, the user could        produce a buzzing sound into a microphone in order to find        sounds like bees or electrical hum.

While DSP analysis may be effective for some groups or classes of songs,it is ineffective for others, and there has so far been no technique fordetermining what makes the technique effective for some music and notothers. Specifically, such acoustical analysis as has been implementedthus far suffers defects because 1) the effectiveness of the analysis isbeing questioned regarding the accuracy of the results, thus diminishingthe perceived quality by the user and 2) recommendations can only bemade if the user manually types in a desired artist or song title, orgroup of songs from that specific website. Accordingly, DSP analysis, byitself, is unreliable and thus insufficient for widespread commercial orother use.

Methods, such as those used by the Muscle Fish patent, which use purelysignal processing to determine similarities thus have problems. Anotherproblem with the Muscle Fish approach is that it ignores the observedfact that often times, sounds with similar attributes as calculated by adigital signal processing algorithm will be perceived as sounding verydifferent. This is because, at present, no previously available digitalsignal processing approach can match the ability of the human brain forextracting salient information from a stream of data. As a result, allprevious attempts at signal classification using digital signalprocessing techniques miss important aspects of a signal that the brainuses for determining similarity.

Previous attempts for classification based on connectionist approaches,such as artificial neural networks (ANN), and self organizing featuremaps (SOFM) have had only limited success classifying sounds based onsimilarity. This has to do with the difficulties in training ANN's andSOFM's. The amount of computing resources required to train ANN's andSOFM of the required complexity are cost and resource prohibitive.

Accordingly, there is a need for an improved method of classifyinginformation that is characterized by the convergence of subjective orperceptual analysis and DSP acoustical analysis criteria to improve theoverall classification efficacy and ease with which music may beretrieved. With such a classification technique, it would be desirableto provide a classification chain, initially formed from a thresholdnumber of training media entities and fine-tuned over time, from whichfurther new media entities may be classified, from which music matchingmay be performed, from which playlists may be generated, from whichclassification rules may be generated, etc.

More particularly, there is a need for a classification chain thatovercomes the limitations of the art by in part using humans to create amap that allows one to uncover relationships between various points inthe attribute space. In essence, it would be desirable to utilize humanexperts to show a classification chain how two points in attributespace, where the attributes are determined by a signal processingalgorithm, relate in perception-space. For instance, two points might bevery close in attribute space, but quite distant in perception space,and thus a proper solution considers and solves this problem in a costeffective manner. In a system that classifies information that ischaracterized by the convergence of subjective or perceptual analysisand DSP acoustical analysis, it would be still further desirable toprovide a system that automatically classifies media entities accordingto tempo properties of at least one portion of an audio file representedby the media entities.

SUMMARY OF THE INVENTION

In connection with a classification system for classifying mediaentities that merges perceptual classification techniques and digitalsignal processing classification techniques for improved classificationof media entities, the present invention provides a system and methodsfor automatically classifying and characterizing tempo properties ofmedia entities. Such a system and methods may be useful for the indexingof a database or other storage collection of media entities, such asmedia entities that are audio files, or have portions that are audiofiles. The methods also help to determine media entities that havesimilar, or dissimilar as a request may indicate, tempo(s) by utilizingclassification chain techniques that test distances between mediaentities in terms of their properties. For example, a neighborhood ofsongs may be determined within which each song has similar tempocharacteristics.

Other features of the present invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The system and methods for providing automatic classification of mediaentities according to tempo properties are further described withreference to the accompanying drawings in which:

FIG. 1 is a block diagram representing an exemplary network environmentin which the present invention may be implemented;

FIG. 2 is a high level block diagram representing the media contentclassification system utilized to classify media, such as music, inaccordance with the present invention;

FIG. 3 is block diagram illustrating an exemplary method of thegeneration of general media classification rules from analyzing theconvergence of classification in part based upon subjective and in partbased upon digital signal processing techniques;

FIGS. 4A through 4D illustrate exemplary aspects of a classificationchain in accordance with the present invention;

FIGS. 5A and 5B illustrate an exemplary calculation of a distance withinwhich two vectors in classification chain input space are considered tobe in the same neighborhood space in accordance with the presentinvention;

FIGS. 6A and 6B illustrate an exemplary process whereby an entry vectoris classified in accordance with other vectors in the classificationchain located within the distance calculated in FIGS. 5A and 5B inaccordance with a classification process of the present invention;

FIG. 6C illustrates an exemplary flow diagram whereby a classificationchain input space is trained for improved classification in accordancewith the present invention;

FIG. 7A illustrates an exemplary formation of a sonic vector accordingto a sonic characterization process of the present invention;

FIG. 7B represents two types of perceptual properties that the soniccharacterization classification chain space of the invention mayclassify;

FIG. 8A illustrates an exemplary flow diagram for a consonancecalculation of the present invention;

FIGS. 8B and 8C illustrate exemplary flow diagrams for a peak detectionand interpolation phase and a peak continuation phase, respectively, formusical consonance and melodic movement calculations in accordance withthe present invention;

FIG. 8D illustrates an exemplary peak intervals calculation phase for amusical consonance calculation in accordance with the present invention;

FIG. 9A illustrates an exemplary flow diagram for a melodic movementcalculation of the present invention;

FIG. 9B illustrates an exemplary melodic vector calculation phase inaccordance with a melodic movement calculation of the present invention;and

FIG. 10 illustrates an exemplary process for extracting tempo propertiesfrom a media entity in accordance with the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Overview

With respect to a classification system for classifying media entitiesthat merges perceptual classification techniques and digital signalprocessing classification techniques, the present invention provides asystem and methods for automatically classifying and characterizingtempo properties of media entities. Tempo may be described as the rateof motion or activity, or as the rate of speed of a musical piece orpassage indicated by one of a series of directions e.g., largo, presto,or allegro.

Such a method and system may be useful in the indexing of a database orother storage collection of media entities, such as audio files, orportions of audio files. The methods also help to determine songs thathave similar, or dissimilar as a request may indicate, sonic propertiesby utilizing classification chain techniques that test distances betweenmedia entities in terms of their properties. For example, a neighborhoodof songs may be determined within which each song has similar sonicproperties.

In exemplary embodiments, the invention includes decimating an inputsignal to a fixed sampling rate, such as a 43 Hz sampling rate. Then,the tempo characteristics of the resulting envelope of data areexaggerated and accentuated, while also smoothing the envelope to removebiases and trends. This may include performing a first order differencecalculation followed by performing half wave rectification and applyinga median filter to smooth out the aberrations, biases and/or trends.Then, after the mean value is subtracted, the data may be half waverectified again. Then, another first order difference calculation may beperformed, followed again by half wave rectification. The resultingwaveform from the above processing is used to generate the final datathat may be input, for example, to a classification chain. The finaldata consists of 130 values or “bins” reflective of the differentcorrelation strengths at different time periods or frequencies. Thefinal data is generated by looping through the decimated waveform andaccumulating the base two logarithm of the product of pairs of pointslocated at a fixed offset from each other. This is done 130 times foroffsets spanning from 1 to 130 samples. The 130 values of the final datamay then be input to a classification chain.

Operation of the classification chain may include two phases:classification and operation. Human experts undertake the classificationphase to provide initial perceptually observed spectral classificationdata to the classification chain. These experts assign each entry in thedata set, to one or more spectral classes, corresponding to somerelevant perceptual spectral or sonic properties of the data. Theclassified data is then used to construct the initial classificationchain. Once an initial classification chain is formed, the operation ofthe classification chain may be observed by human experts, and furthertrained for improved performance. Once the classification chain is readyfor operation, it may be used to classify or otherwise relate songsaccording to their sonic properties.

Exemplary Computer and Network Environments

One of ordinary skill in the art can appreciate that a computer 110 orother client device can be deployed as part of a computer network. Inthis regard, the present invention pertains to any computer systemhaving any number of memory or storage units, and any number ofapplications and processes occurring across any number of storage unitsor volumes. The present invention may apply to an environment withserver computers and client computers deployed in a network environment,having remote or local storage. The present invention may also apply toa standalone computing device, having access to appropriateclassification data and an appropriate playlist generation engine.

FIG. 1 illustrates an exemplary network environment, with a server incommunication with client computers via a network, in which the presentinvention may be employed. As shown, a number of servers 10 a, 10 b,etc., are interconnected via a communications network 14, which may be aLAN, WAN, intranet, the Internet, etc., with a number of client orremote computing devices 110 a, 110 b, 110 c, 110 d, 110 e, etc., suchas a portable computer, handheld computer, thin client, networkedappliance, or other device, such as a VCR, TV, and the like inaccordance with the present invention. It is thus contemplated that thepresent invention may apply to any computing device in connection withwhich it is desirable to provide classification services for differenttypes of content such as music, video, other audio, etc. In a networkenvironment in which the communications network 14 is the Internet, forexample, the servers 10 can be Web servers with which the clients 110 a,110 b, 110 c, 110 d, 110 e, etc. communicate via any of a number ofknown protocols such as hypertext transfer protocol (HTTP).Communications may be wired or wireless, where appropriate. Clientdevices 110 may or may not communicate via communications network 14,and may have independent communications associated therewith. Forexample, in the case of a TV or VCR, there may or may not be a networkedaspect to the control thereof. Each client computer 110 and servercomputer 10 may be equipped with various application program modules 135and with connections or access to various types of storage elements orobjects, across which files may be stored or to which portion(s) offiles may be downloaded or migrated. Any server 10 a, 10 b, etc. may beresponsible for the maintenance and updating of a database 20 inaccordance with the present invention, such as a database 20 for storingclassification information, music and/or software incident thereto.Thus, the present invention can be utilized in a computer networkenvironment having client computers 110 a, 110 b, etc. for accessing andinteracting with a computer network 14 and server computers 10 a, 10 b,etc. for interacting with client computers 110 a, 110 b, etc. and otherdevices 111 and database(s) 20.

Classification

In accordance with one aspect of the present invention, a uniqueclassification technique is implemented which combines human and machineclassification techniques in a convergent manner, from which aclassification chain, which embodies a canonical set of rules forclassifying music, may be developed, and from which a database, or otherstorage element, may be filled with the classification chain and/orclassified songs. With such techniques and rules, radio stations,studios and/or anyone else with an interest in classifying music will beenabled to classify new music. With such a database, music associationmay be implemented in real time, so that playlists or lists of related(or unrelated if the case requires) media entities may be generated.Playlists may be generated, for example, from a single song and/or auser preference profile in accordance with an appropriate analysis andmatching algorithm performed on the data store of the database. Nearestneighbor and/or other matching algorithms may be utilized to locatesongs that are similar to the single song and/or are suited to the userprofile. Based upon a distance measurement from the mean, median, etc.of a certain class in the classification chain, a confidence level forsong classification may also be returned.

FIG. 2 illustrates an exemplary classification technique in accordancewith the present invention. Media entities, such as songs 210, fromwherever retrieved or found, are classified according to humanclassification techniques at 220 and also classified according toautomated computerized DSP classification techniques at 230. 220 and 230may be performed in either order, as shown by the dashed lines, becauseit is the marriage or convergence of the two analyses that provides astable set of classified songs at 240. As discussed above, once such adatabase of songs is classified according to both human and automatedtechniques, the database becomes a powerful tool for generating songswith a playlist generator 250. A playlist generator 250 may takeinput(s) regarding song attributes or qualities, which may be a song oruser preferences, and may output a playlist, recommend other songs to auser, filter new music, etc. depending upon the goal of using therelational information provided by the invention. In the case of a songas an input, first, a DSP analysis of the input song is performed todetermine the attributes, qualities, likelihood of success, etc. of thesong. In the case of user preferences as an input, a search may beperformed for songs that match the user preferences to create a playlistor make recommendations for new music. In the case of filtering newmusic, the rules used to classify the songs in database 240 may beleveraged to determine the attributes, qualities, genre, likelihood ofsuccess, etc. of the new music.

In accordance with the present invention, once the classification chaindatabase 240 takes on a critical mass, defined as the processing ofenough media entities to form a reasonably valid rule set andcorresponding song database 240 within a threshold tolerance, playlistgenerator 250 may be a powerful tool for training new humans. Forexample, if a particular human is having difficulty learning a certainconcept, playlists may be formed that emphasize (or de-emphasize) theeffect to illustrate the concept in greater depth to a trainee.Naturally, at the outset, before such critical mass is reached, anotherplaylist generator or manually formed playlists may be utilized. Thetraining process of the present invention is described in more detailbelow. In effect, the rules can be used as a filter to supplement anyother decision making processes with respect to the new music.

FIG. 3 illustrates a process that generates generalized rules for aclassification system characterized by the convergence of subjective orperceptual analysis and DSP acoustical analysis criteria. A first goalis to train a database with enough songs so that the human and automatedclassification processes converge, from which a consistent set ofclassification rules may be adopted, and adjusted to sufficientaccuracy. First, at 305, a general set of classifications are agreedupon in order to proceed consistently i.e., a consistent set ofterminology is used to classify music in accordance with the presentinvention. At 310, a first level of expert classification isimplemented, whereby experts classify a set of training songs indatabase 300. This first level of expert is fewer in number than asecond level of expert, termed herein a groover, and in theory hasgreater expertise in classifying music than the second level of expertor groover. The songs in database 300 may originate from anywhere, andare intended to represent a broad cross-section of music. At 320, thegroovers implement a second level of expert classification. There is atraining process in accordance with the invention by which grooverslearn to consistently classify music, for example to 92-95% accuracy.The groover scrutiny reevaluates the classification of 310, andreclassifies the music at 325 if the groover determines thatreassignment should be performed before storing the song in humanclassified training song database 330. The present application describesthis training process for training humans to recognize fundamentalproperties of media entities in greater detail below.

Before, after or at the same time as the human classification process,the songs from database 300 are classified according to digital signalprocessing (DSP) techniques at 340. Exemplary classifications for songsinclude, inter alia, tempo, sonic, melodic movement and musicalconsonance characterizations. Classifications for other types of media,such as video or software are also contemplated. The quantitativemachine classifications and qualitative human classifications for agiven piece of media, such as a song, are then placed into what isreferred to herein as a classification chain, which may be an array orother list of vectors, wherein each vector contains the machine andhuman classification attributes assigned to the piece of media. Machinelearning classification module 350 marries the classifications made byhumans and the classifications made by machines, and in particular,creates a rule when a trend meets certain criteria. For example, ifsongs with heavy activity in the frequency spectrum at 3 kHz, asdetermined by the DSP processing, are also characterized as ‘jazzy’ byhumans, a rule can be created to this effect. The rule would be, forexample: songs with heavy activity at 3 kHz are jazzy. Thus, when enoughdata yields a rule, machine learning classification module 350 outputs arule to rule set 360. While this example alone may be anoversimplification, since music patterns are considerably more complex,it can be appreciated that certain DSP analyses correlate well to humananalyses.

However, once a rule is created, it is not considered a generalizedrule. The rule is then tested against like pieces of media, such assong(s), in the database 370. If the rule works for the generalizationsong(s) 370, the rule is considered generalized. The rule is thensubjected to groover scrutiny 380 to determine if it is an accurate ruleat 385. If the rule is inaccurate according to groover scrutiny, therule is adjusted. If the rule is considered to be accurate, then therule is kept as a relational rule e.g., that may classify new media.

The above-described technique thus maps a pre-defined parameter space toa psychoacoustic perceptual space defined by musical experts. Theprocess whereby people are trained to be or certified as “musicalexperts,” for purposes of uniformly applying classification techniquesis the subject of the present application.

Adaptive Media Property Classification

The present invention relates to a system and methods for automaticallyclassifying data according to perceptual properties of the data and tomethods for generating and utilizing a classification chain. Theclassification chain is suited to the searching and sorting of largedatabases of sensory data, including, but not limited to, music, imageand other media databases.

The operation of the classification chain is broken down into twophases: classification and operation. Human experts undertake theclassification phase. U.S. patent application Ser. No. 09/934,774describes a system and method for training and certifying trainees to begroovers, or experts qualified to classify data. These experts, who maybe first-rate music classification experts for maximum consistency, orgroovers who have been trained by those experts, assign each mediaentity in the data set to one or more classes. Each class corresponds toa given subset of perceptual properties of the data. The classified datais then used to construct an initial classification chain.

In an exemplary non-limiting embodiment, the fundamental properties ofmedia entities, such as songs, are grouped into three main areas:rhythm, zing and mood. Rhythm may include tempo, time signature, rhythmdescription, rhythm type and rhythmic activity. In the case of mood, thesub-categories may include emotional intensity, mood and mooddescription. In the case of zing, the sub-categories may includeconsonance, density, melodic movement and weight. Once a trainee learnsthe nature of and how to recognize distinctions for these terms, atrainee becomes a groover and may classify new songs or song segments.

In an exemplary non-limiting embodiment for the classification of newsongs or song segments, a groover enters values for attributes for thesong or song segment including song-level attributes and voice-levelattributes. Some of these attributes are similar to the fundamentalproperties described above. Song-level attributes may include tempo,weight, harmony, intensity, density, consonance, chordal movement, mood,range, flow, dynamics, rhythm description, rhythm type, rhythmicactivity, rhythm flexibility, rhythm time signature and description.Voice-level attributes include prominence, melodic movement, intensity,ornamentation, cleanliness, rhythm activity and whether the song has asolo. Values may be entered via discrete adjective descriptors, viacontinuous or discrete range(s) of numbers, via more subjective termslike, low, medium and high, jagged or smooth, and the like, as long asthe classification values used are consistent from segment to segment orfrom song to song.

FIGS. 4A through 4D illustrate exemplary aspects of a classificationchain as implemented in an embodiment of the present invention. FIG. 4A,for instance, illustrates that an expert or groover assigns to a mediaentity ME during classification at least one classified-by-human valuefor various categories or classes CH11, CH12, . . . , CHMN that describethe media entity, such as values for ranges or descriptions ofsong-level and voice-level attributes (such as those noted in exemplarydescription above). FIG. 4B similarly illustrates that computingdevice(s) also assign at least one classified-by-computer value forvarious categories or classes CC11, CC12, . . . , CCMN that describe DSPcharacteristics of the media entity ME. Some of these DSPcharacteristics and techniques for automatically generating thosecharacteristics are described in more detail below. These includemeasurements of sonic characterization, musical consonance, tempo andmelodic movement and are well suited to describing media entities, suchas songs.

FIG. 4C illustrates an exemplary vector for the media entity ME aftersaid at least one classified-by-human value(s) are assigned and aftersaid at least one classified-by-computer value(s) are assigned. Thevalues assigned in FIGS. 4A and 4B become part of the vector for themedia entity ME. The at least one classified-by-human value(s) and theat least one classified-by-computer value(s), with any other relevantdescriptive or classification information or values, are placed in avector V_(ME), which may then be accommodated in the classificationchain input space, such as classification chain input space ISillustrated in FIG. 4D. With the classification chain input space IS, anew unclassified entry vector EV1 may be input to the space IS, andvarious searching, matching and/or analyzing of the entry vector EV1relative to the vectors of the classification chain input space IS maybe performed. For example, if another vector is within a distance d_nhof the entry vector EV1, the other vector may be considered within theentry vector EV1's neighborhood, and further calculations and analysismay be based on the vectors contained within EV1's neighborhood. Otherdistances and/or properties may be useful for analyzing the entry vectorEV1 relative to the other vectors in classification chain input space ISas well.

A neighborhood is defined as the region in the input space that is“close” in distance to the point specified by the entry vector. In thisregard, distance may be defined as the Euclidian distance: |A−B|, whereA and B are vectors, although any distance measurement utilized by thoseof ordinary skill in the art may be used. FIGS. 5A and 5B illustrate anexemplary calculation of a distance within which two vectors inclassification chain input space are considered to be in the sameneighborhood space. FIG. 5A illustrates a simplified classificationchain input space IS that contains nine (9) vectors CCV1 through CCV9.For purposes of operation of the present invention, the distance withinwhich two points are considered to be in the same neighborhood isdetermined in the following exemplary manner. First, the nearest entriesin the classification chain to several entry vectors EV1, EV2 and EV3are determined. The several entry vectors EV1, EV2 and EV3 are notmembers of the classification chain, but have been classified by thehuman experts. In the example, EV1 is closest to CCV1 at distance d1,EV2 is closest to CCV4 at a distance d4 and EV3 is closest to CCV6 atdistance of d6. Then, the distances d1, d4 and d6 between the chainentries and entry vectors EV1, EV2 and EV3 are calculated and/or stored.Then, the class(es) of the nearest entries CCV1, CCV4 and CCV6 areassigned to the input entry vectors EV1, EV2 and EV3, respectively.Then, the classification error, defined as the difference between theclassification chain's class(es) estimate and the entry vectors' EV1,EV2 and EV3 class(es) as determined by a human expert, are calculatedfor each entry vector.

Then, as illustrated in FIG. 5B, a histogram is generated illustratingthe classification error for each of the nearest neighbor distances. Themaximum distance at which the classification error is acceptable is theneighborhood distance d_nh. In the example shown, e2 is acceptable errorin classification and e4 is unacceptable error in classification. Eitherof distances d1 and d4 could be chosen as a neighborhood distance, orsome distance between d1 and d4 could be chosen as well. For example, inan exemplary embodiment, the linear interpolation between the leastacceptable, but still acceptable classification error and leastobjectionable, but still objectionable classification error isdetermined. In this example, such interpolation means that e3 is thethreshold error that is allowed for same neighborhood vectors, and d_nhis the neighborhood distance, the distance within which two separatevectors may be considered to be within the same neighborhood.

Thus, once enough vectors describe the classification chain input spaceIS and the neighborhood distance is determined, the classification chainis ready for operation. Thus, once the classification chain is generatedwith sufficient breadth in representation of media entity classificationinformation, the operation phase for the classification chain may begin.During operation, when presented with an unclassified entry such as anentry vector EV, the classification chain returns an estimate of theclass of the entry, as well as a confidence measure that is proportionalto the level of confidence of the class assignment. For a simplifiedexample, a classification chain can be constructed to determine thetempo of digitally recorded songs. The new song data is presented to thechain and the chain returns an estimate of the song's tempo as well as anumber corresponding to the degree of certainty of the tempo estimate;the larger the confidence number, the more certain the chain is of theclassification. If the confidence for the tempo determination is low, anexpert or groover may be called upon to find out why. An expert may, forexample, determine that a new tempo class should be constructed toaccommodate music of the type that generated the tempo estimate of lowconfidence. Alternatively, the expert may determine that the music maybe of a sort that does not lend itself easily to tempo characterization,and thus tempo characterization may not be suitable for that type ofmusic. Other unforeseen aspects of tempo characterization may arise aswell. In short, a human may have more insight as to why theclassification chain fell short of determining the tempo class with highaccuracy or confidence. Over time, as new classes are added to theclassification chain or as previous classes are modified, theclassification chain becomes a more effective and powerful tool forquickly characterizing new and existing music.

FIGS. 6A and 6B illustrate an exemplary process whereby an entry vectorEV is classified in accordance with vectors of CCV1 through CCV9 in theclassification chain input space IS that are located within the distanced_nh calculated in FIGS. 5A and 5B. When a new media entity is input tothe classification chain for classification, the estimated classreturned by the classification chain is determined by calculating therepresentative class of the neighborhood in which the entity falls.Thus, in the example, CCV2, CCV4, CCV6 and CCV9 are located in entryvector EV's neighborhood because CCV2, CCV4, CCV6 and CCV9 are withindistance d_nh, as calculated above.

The input data of entry vector EV may be treated as a vector of Ndimensions where N is the number of discrete elements in the data. For adigitally recorded song presented to the classification chain asrecorded samples in PCM format, N could be on the order of severalmillion. In many cases, the amount of relevant data can be reduced to asmall fraction of that number without affecting the performance of theclassification engine. For example, as described below, digital signalprocessing measurements of sonic characterization, musical consonance,tempo and melodic movement may be made with reduced amounts of data.

When in operation mode, entries such as entry vector EV are presented tothe classification chain and an estimate of the entry class is returnedalong with a measure of certainty. If the classes in the classificationchain come from discrete sets, the assigned class is the median classvalue of all classification chain entries that fall within the inputentry's neighborhood, where neighborhood is defined above. Theconfidence value is the number of classification chain entries withinthe neighborhood with the median class divided by the number of entriesin the neighborhood. In the example of FIG. 6B, CCV2, CCV4, CCV6 andCCV9 are in entry vector EV's neighborhood and at distances s2, s1, s3and s4, respectively. If CCV2, CCV4 and CCV6 classify EV as having afast tempo, and CCV9 classifies EV as having a medium tempo, then theclassification chain classifies EV as having a fast tempo with 75%confidence. If the classification chain is used to classify a member ofa continuous set or range, then the class assigned to media entity isthe neighborhood mean. The confidence level is inversely proportional tothe standard deviation of the neighborhood values. For example, if CCV2classified EV as having a tempo of 2, CCV4 classified EV as having atempo of 3, CCV6 classified EV as having a tempo of 3 and CCV9classified EV as having a tempo of 4, then the assigned class is a tempoof 3, and the confidence percentage is calculated as a function of thevariance i.e., the standard deviation of the numbers 2, 3, 3 and 4.

If the confidence level of an input entry is low, the input entry issent to a human expert for classification after which, it may be addedto the classification chain. In this manner, the classification chainadapts to new data. Precisely what constitutes low must be determinedfor an application or type of media entity.

The ratio of the number of entries added to the classification chain tothe total number of entries presented to the chain is tracked during theoperation process to determine if the classification process issucceeding. Ideally, this number should approach zero asymptotically.This assumes that the human classification is without error and that therelevant information is contained in each entry so that classificationcan succeed. In reality, the ratio eventually converges to some numberless than one, but greater than zero. The more successful theclassification, the lower the ratio.

FIG. 6C illustrates an exemplary flow diagram whereby the classificationchain input space is trained, and “tweaked” over time for improvedclassification. This process could be performed for a single property,such as tempo, or for a plurality of properties up to the total numberof classes and subclasses for the media entity. Having an initialclassification chain with which to work as a result of human expertclassification of songs, an exemplary fine tuning process utilizing aplurality of unclassified media entities may proceed as follows: Atstart 600, a first unclassified song is presented. If, at 605, it isdetermined via a fingerprint or other identification means that the songor portion of the song is already in the database, such as database 240,then the flow proceeds to 610. If at 610, training is complete eitherbecause an expert determines that the classification chain is workingwith threshold accuracy or because the last of the plurality ofunclassified songs has been processed, then the process ends at 625. Iftraining is not complete, the next song is presented at 615 and the flowreturns to 605. If at 605, it is determined that the song has not yetbeen entered into the data set or previously classified, a song vectoris calculated for the desired properties of the song at 620, and thevector is presented to the classification chain. The classificationchain returns a response at 630 i.e., the classification chain returns aclass estimate for the input vector and a degree of confidence for theestimate. If there is a high level of confidence for the classificationchain's ability to classify the song, there is no need to tweak theclassification chain because it worked for its intended purpose, and thenext song of the plurality of unclassified songs at 645 is ready to beprocessed and the process begins again at 605. If the degree ofconfidence for the classification is low or if an expert determines thatthere is otherwise a problem with the classification of the song, thenthe flow proceeds to 640 where the expert assigns values to the song,and then adds the newly classified song to the classification chain forfuture classifications. In this case, in the future, when anunclassified song is input to the classification chain that is similarto the song classified by the expert at 640, the classification chainwill produce a better response than the previous response because it hasbeen further trained to recognize songs of that character.

The present invention provides a number of clear advantages over theprior art. For example, a computing device in accordance with thepresent invention allows a user to use digital signal processing methodsto determine similarities among sounds as judged by a trained humanexpert. This is in contrast to other methods referenced in thebackground which use purely signal processing to determine similarities.As mentioned, the main problem with those approaches is that they ignorethe observed fact that often times, sounds with similar attributes ascalculated by a digital signal processing algorithm will be perceived assounding very different. This is because, at present, no previouslyavailable digital signal processing approach can match the ability ofthe human brain for extracting salient information from a stream ofdata. As a result, all previous attempts at signal classification usingdigital signal processing techniques alone miss important aspects of asignal that the brain uses for determining similarity.

The classification chain of the present invention overcomes thislimitation by using humans to create a map that allows one to uncoverrelationships between various points in the attribute space. In essence,human experts are used to show the classification chain how two pointsin attribute space, where the attributes are determined by a signalprocessing algorithm, relate in perception-space. Two points might bevery close in attribute space, but quite distant in perception space,and the present invention identifies this distinction because perceptualand attribute space are correlated.

Current implementations of the classification chain show that thecorrelation mapping is indeed quite complex, which in the case of songs,may require tens of thousands of human classified entries to build areliable classification chain. This degree of complexity would beextremely difficult to implement in a digital signal processingapplication without the aid of human training. The present inventionalso avoids the prohibitively computationally intensive training phasesof SOFMs and ANNs. Therefore, the classification chain requires muchless processing time before it is ready for processing and classifyingdata.

Also advantageously, the structure of the classification chain of theinvention need not be specified before operation. The classificationchain of the invention grows in accordance with its performance, onlymodifying entries or adding new entries when needed. This is in contrastto hitand-miss approaches used when designing ANNs and SOFMs.

Method and System for Sonic Characterization

One application for the above-described classification chain is withrespect to a measurement of perceived sonic properties of sound. In oneaspect, the present invention thus provides a method for automaticallyclassifying and characterizing music according to sonic properties ofthe media entities. The sonic properties represent the perceivedlong-term properties of the sound and include various aspects of thesound's timbre, including such notions as: spectral balance, spectralweight of the sound and the perceived spectral density. Spectral balanceis a relative measurement of bass energy versus treble energy, i.e.,does the sound have more bass energy than treble energy? Spectral weightis a relative measurement of the perceived quality of the sound i.e., isthe sound perceived as ‘heavy’ or ‘light’? The perceived spectraldensity is a relative measurement of how much of the spectrum isperceived as being used. Thus, sonic characterization includesmeasurements of various aspects of the information content of mediaentities. Such a method and system may be useful in the indexing of acatalog of sounds, which could be, for example, a collection of soundeffects or a music database, such as database 240. The method also helpsto determine the sonic similarity between different sounds by utilizingthe above-described classification chain techniques. For example, aneighborhood of songs may be determined within which each song has asimilar sonic characterization.

The operation of sonic characterization may include a construction phaseand a classification phase. During the construction phase, human expertsclassify a representative set of sounds according to their perceptualqualities. The experts assign to each entry in a representative set ofmedia entities a value or set of values for the perceived spectralqualities of the media entities. After the classification is completed,each sound in the representative data set is reduced to a set ofcharacteristic numbers, referred to as the sound's characteristicvector. When the characteristic vectors of the representative mediaentities are added to the classification chain input space, newunclassified media entities may be classified based upon media entitiesfound in their neighborhood of the classification chain input space.

The characteristic vector(s) are calculated in the following manner: Thesound is first broken up into a plurality of frames, each framecomprised of a fixed number of pulse code modulation (PCM) values, eachvalue representing a sample in the frame. PCM is a digital scheme fortransmitting analog data. The signals in PCM are binary and thusrepresented either by logic 1 (high) and logic 0 (low). Using PCM, it ispossible to digitize all forms of analog data, including full-motionvideo, voices, music, telemetry, virtual reality (VR) and others.

While the present invention works with any format of music data such as.wav, .mp3, .rp, etc., it should also be understood that the inventionworks with analog data as well since analog data may be converted todigital data. For example, as one of ordinary skill in the art canappreciate that to obtain PCM from an analog waveform at the source(transmitter end) of a communications circuit, the analog signalamplitude is sampled (measured) at regular time intervals. The samplingrate, or number of samples per second, is generally chosen to be severaltimes the maximum frequency of the analog waveform in cycles per second,or hertz. The instantaneous amplitude of the analog signal at eachsampling is rounded off to the nearest of several specific,predetermined levels. This process is called quantization and isgenerally achieved with a power of 2—for example, 8, 16, 32, or 64quantization levels with respective 3, 4, 5 and 6 quantization bits. Theoutput of a pulse code modulator is thus a series of binary numbers,each represented by some power of 2 bits.

Additionally, it can be appreciated that any digital format may beconverted back to analog format. For example, in the case of a PCMsignal, at the destination (receiver end) of the communications circuit,a pulse code demodulator, having the same number of quantum levels aspossessed by the modulator, may convert the binary numbers back intopulses. These pulses may be further processed to restore the originalanalog waveform.

Thus, in accordance with an exemplary embodiment describing thecalculation of a media entity's characteristic sonic vector, the soundis first broken up into a plurality of frames, with each frame comprisedof a fixed number of PCM values, and with each PCM value representing asample in the frame. For each frame, the energy of the frame iscalculated by calculating the root mean squared (RMS) value of theframe. An FFT of that frame is then taken. The entropy content of theframe is then calculated by normalizing the sum of the magnitude of theFFT to one, and then calculating:$S = {- {\sum\limits_{w}^{\quad}\quad{p_{w}{\log_{2}\left( p_{w} \right)}}}}$where S is the entropy of the frame, p_(w) is the normalized magnitudeof bin w of the FFT, and log₂(p_(w)) is the log base 2 of (p_(w)). Theenergy in each of several critical bands is also calculated by summingthe value of the square of the magnitude of each FFT bin that fallswithin a given critical band. Measurements are also taken of thederivative of each of the critical band filtered waveforms to gaininformation about the amount or rate of change of the frequenciesrepresented by the frames of the media entity. The entropy content,derivative and the energy values are calculated for each frame of themedia entity. This information then becomes the bases for the soniccharacterization vector.

As mentioned, the human auditory system has a limited,frequency-dependent resolution and thus sonic characterization thatincludes a filter to account for this fact yields better results. Thisis known generally as critical band filtering. A more perceptuallyuniform measure of frequency may thus be expressed in terms of the widthof the critical bands. These critical bands have less than a 100 Hzwidth at the lowest audible frequencies, and a width of more than 4 kHzat the high end of the perceptually observable frequency spectrum. Theaudio frequency range for purposes of perceptual human analysis, forexample, can be partitioned into about 25 critical bands.

FIG. 7A illustrates an exemplary formation of a sonic vector accordingto the sonic characterization process of the present invention. At 745,a media entity is received by the system and the data is converted fromthe time domain to the frequency domain via a Fast Fourier Transform(FFT). The FFT is performed on the frame data to produce a raw digitalrepresentation of the spectral characteristics of the media entity.Subsequently, each frame may be processed in the following manner. Foreach frame of datalist 750, critical band filtering is performed on thedata, and the average of the data is calculated at 765. The derivativeof the critical band filtered data is also taken at 760, and thederivative data is also averaged at 765. In parallel to 750 and 760, at755, an entropy calculation according to the above-described equation isperformed for each frame. These values too are averaged at 765. In anexemplary embodiment, 12 values result from 12 critical band filteringdata sets, 12 values result from a corresponding 12 derivativecalculations from the 12 critical band filtering data sets, and I valuefor the entropy form the feature vector at 770. Principal ComponentAnalysis (PCA) may optionally be performed at 775 on the feature vectordata in order to extract the orthogonal or most salient features fromthe data in terms of what it represents. The feature vector may then beadded to the classification chain at 780. Once enough feature vectorsare added to the classification chain, the classification chain is readyfor operation.

In addition to the calculation of the mean of each value at 765, thestandard deviation of each value may also be calculated. The resultingvalues are the numbers that comprise the characteristic vector for thesound data presented. These values are then stored in a classificationchain for use as described above in the Adaptive Media PropertyClassification section.

Thus, during operation of the classification chain, when an unknownsound is presented to the device for classification, its characteristicvector is calculated and presented to the classification chain mentionedabove. The classification chain then returns an estimate of the spectralproperties of the sound data, as well as confidence level for thosespectral properties.

The described system and method allows the user to employ digital signalprocessing methods to determine the perceived sonic properties, in amanner that simulates the results that would be assigned by a trainedhuman expert. This is in contrast to other approaches that use moretraditional digital signal processing techniques to classify sonicattributes of a particular sound. By using a large collection of datathat has been classified by trained experts, an approximation to thecomplex processing function performed by the brain is obtained. As moredata is presented, the system and techniques improve their ability toclassify, as results that are returned from the classification chainwith a low confidence are categorized by humans and then entered intothe classification chain. This results in a dynamic system that is ableto improve performance over time.

FIG. 7B represents two types of perceptual properties that the soniccharacterization classification chain space may classify: mood andemotion. Intuitively, when listening to a song, a listener or expert canassign a relative happiness or sadness to the music. This describes themood of the song. Similarly, even within a mood class, a listener orexpert could assign an intensity to the happiness or sadness from low tohigh. For the same reason, an emotionally intense song could rangeanywhere from sad to happy. The sonic characterization classificationchain input space correlates well to these properties. It is of importin this regard that spectral changes weigh into sonic characterization.

Method and System for Musical Consonance Classification

One application for the above-described classification chain is withrespect to a measurement of perceived consonance of sound. Consonance isa measurement of the perceived harmony or agreement among components ofan audio entity, and generally relates to the correspondence orrecurrence of sounds. In one aspect, the present invention thus providesa method for automatically classifying and characterizing musicalconsonance.

Such a method and system may be useful in the indexing of a catalog ofsounds, which could be, for example, a collection of sound effects or amusic database, such as database 240. The method also helps to determinesongs having similar or dissimilar consonance by utilizing theabove-described classification chain techniques. For example, aneighborhood of songs may be determined within which each song has asimilar consonance.

As illustrated in FIG. 8A, after performing an FFT on a data entry at800, the invention includes a peak detection and interpolation phase802, a scalable critical band masking or filtering phase 804, a peakcontinuation phase 806, an intervals or consonance calculation phase 808and a classification phase 814. A feature vector 810 is extracted fromthe consonance calculation phase 808, and optionally PCA may beperformed on the feature vector at 812 to extract the salientinformation represented by the vector data. An audio entity, such as asong represented by raw PCM audio data, is read into the peak detectionand interpolation stage where the most prominent peaks in the frequencydomain along with their energies are detected and recorded into outputmatrices. These matrices are then fed through the scalable critical bandmasking stage, the peak continuation stage, and then into the intervalscalculation stage where the frequency of ratios between peaks are storedinto a final output vector for each sound. This vector is then fed intothe classification chain which interprets the output vector and whichmay assign a consonance value for the sound.

Peak detection 802 is performed on a frame-by-frame basis of an audiofile by recording the locations and energies of the peaks in thespectrum for each frame. The top P number of most energetic peaks areselected for each frame and recorded with their respective energy intooutputs vectors for each frame, where P is a pre-defined parameter.These peak energies and locations are then Nth-order interpolated toincrease precision. The final output is two P×F matrices, where F is thenumber of frames: one holding the P maximum peak locations (frequency inHz) for each frame, and the other holding the respective energy valuefor the peak location in each frame.

The peak detection and interpolation phase 802 may be described in moredetail with reference to FIG. 8B illustrating a flow diagram of someexemplary non-limiting pseudocode which one of ordinary skill in the artmight implement among many different software or firmware realizationsof the present invention. At 816, a loop is set for a current framenumber to run from the first to last frame for the audio segment. At818, zero-padding is performed as necessary to create uniform framelength, and the FFT of the data is performed. At 820, an estimate of thederivative of the FFT frame is calculated by storing the value of thedifference between adjacent bins in the given frame. This frame isreferred to as the difference frame. At 822, a new loop from 1 to thenumber of bins in the difference frame is started. For each location inthe difference frame, if the value at that location is greater thanzero, as determined at 824, and the value at the next location is lessthan zero, as determined at 826, then the bin at the location and itsenergy is recorded into the output matrices at 828. If either the valueat that location is not greater than zero, as determined at 824 or thevalue at the next location is not less than zero, as determined at 826,then the loop is repeated for the next location in the difference framewithout recording into the output matrices, until the loop is performedfor the last bin in the difference frame.

Thus, this determination is repeated for each bin in the differenceframe until the last bin is reached at 830, at which point all peaks inthe FFT frame have been identified. At 834 the number of requested peaksare identified. Then, at 836, another loop is set to run from 1 to thenumber of peaks requested. At 838, an Nth order interpolation of thepeaks' positions and heights is performed to increase the accuracy ofthese values until the number of peaks requested is reached at 840. Whenall of the peaks for the number of peaks requested have been Nth orderinterpolated, the processing for the next frame begins at 842 until allof the frames have been processed.

The scalable critical band masking phase 804 filters and removes anypeaks that are masked by surrounding peaks with more energy. The amountof masking is scalable, and this feature can be turned off completely.Thus, this is a technique whereby the significance of peaks having lessenergy than surrounding peaks with greater energy can be eliminated orreduced.

The peak continuation phase 806 is performed by keeping track of peaksthat last for more than a specified number of frames, and by filling inpeaks where they may have been missed for one or more instances in time.An exemplary implementation of a peak continuation process in accordancewith the present invention is performed at 868, described in more detailbelow. This is accomplished by using peak ‘guides’ that are initiallyset to the peak locations in the first frame, and then on aframe-by-frame basis, finding a suitable guide for each peak within theframe. If a guide is found, then the peak-energy data is saved andcontinued, otherwise the data is discarded.

The peak continuation phase 806 may be described in more detail withreference to FIG. 8C illustrating a flow diagram of some exemplarynon-limiting pseudocode which one of ordinary skill in the art mightimplement among many different software or firmware realizations of thepresent invention. At 846, initial guides are set in place based on thepeaks found in the first frame. Then, a loop is started at 848 to cyclethrough the number of frames set for the media entity, such as a song.At 850, a peakmatched vector is initially set to a null set, andguidematch is set to zero. At 852, another loop is started to cyclethrough the number of peaks in the current frame. Next, at 854, avariable MinDiff is defined as a constant k times the Nyquist Frequency,corresponding to the signal's sampling rate. Matched is initiallyassigned zero. A ThisPeak value is set to the bins matrix values at thecurrent frame number and current peak number. A ThisEnergy value is setto the energy matrix values at the current frame number and current peaknumber. Then, at 856, another loop is started to cycle through the guidenumbers. At 858, a variable ThisGuide is set to the guide frequency forthe current guide number and a variable ThisGuideEnergy is set to theguide energy of the current guide number. Once these values are set fora given frame number, a given peak number and a given guide number, at860, (i) if the ThisPeak matrix value is within ½ step of ThisGuide,where a value is within ½ step when that value is withinThisGuide×2^(±1/12), (ii) the ThisEnergy matrix value is within athreshold logarithmic distance, such as 6 dB, of ThisGuideEnergy and(iii) ThisGuide was not just continued, then flow proceeds to 862. Inthis regard, in an exemplary non-limiting embodiment, if the state ofThisGuide is 2, then ThisGuide was just continued. If, at 860, theThisPeak matrix value is not within ½ step of ThisGuide or theThisEnergy matrix value is not within the threshold distance ofThisGuideEnergy or ThisGuide was just continued, then flow proceeds to864.

At 862, Guidematch is assigned to the current guide number to record thematched guide number as determined at 860. At 864, the current guidenumber is increased and 858 to 864 are performed until the guides havebeen tested against the current peak in the frame. Thus, if a guidecorresponded to the current peak during the cycle, at 866, guidematchwill be non-zero and 868 will be performed. If, however, no guidescorresponded to the peak during the cycle, the flow proceeds to 870 tomove on to the next peak, and processing thereof beginning at 854. At868, the peak that corresponded to the guide has its frequency andenergy recorded into an output matrix. In addition, any possibly missedpeaks in the previous frame are recorded if ThisGuide was just started.The PeakMatched array is assigned 1 at the peak number position. Also, aGuideState array and GuideEnergy array for the guide number position areassigned to the number of times the guide has continued and to theThisEnergy value, respectively. At 870, if all of the peaks for theframes have been processed, the flow proceeds to 872, where unusedguides and unmatched peaks are located and recorded. If, at 874, thereare any unused guides or unmatched peaks, then, at 876, new guides arestarted at the unmatched peaks with the largest energy and the flowproceeds to 878. If not, flow proceeds directly to 878. At 878, the nextframe is made ready for processing to repeat the procedure beginning at850, until all frames have been processed and the algorithm finishes at879.

An alternative procedure for implementing the peak continuation processrepresented by 868 includes (i) at the current frame, recording thispeak and energy into an output matrix, (ii) if this guide was juststarted, e.g., guide's active state equals zero, then search for a peakmatch, e.g., similarly to 860, at some pre-specified number of framesprevious to the current frame, for instance, 2 frames previous to thecurrent frame and (iii) if there is a match, then record this peak andenergy in all frames between the current frame and the frames that havebeen gone back over, i.e., the previous frames that have been analyzedfor missed peaks.

The peaks intervals calculation stage 808 creates a vector, e.g., a 1×24output vector, containing the mean energy of the ratio between peaks forall frames. This vector is created by binning the number of occurrencesof ‘ratio’ (when less than two octaves, or 25) in the followingequation:ratio=nearest_integer(12*log2(peak 1/peak2))

All peaks within each frame are compared to others in that frame, andthis is done for all frames. Finally, the “intervals” vector is dividedby the number of total frames to get the mean value for each ratio.Additional values beyond two octaves may be wrapped to the two octavesas if within the two octaves for purposes of calculation.

The peak intervals calculation phase 808 may be described in more detailwith reference to FIG. 8D illustrating a flow diagram of some exemplarynon-limiting pseudocode which one of ordinary skill in the art mightimplement among many different software or firmware realizations of thepresent invention. At 880, a FrameNum variable is set to loop from 1 tothe number of frames. At 882, a PeakBottom variable is set to loop from1 to the number of peaks in the frame. At 884, a denominator value isset to the peak location matrix value at the current frame andPeakBottom value. At 886, if the denominator value is non-zero then, at888, a PeakTop variable is set to loop from 1 to the number of peaks aswell. If the denominator is zero, then the flow proceeds to 900. At 890,a numerator value is set to the peak location matrix value at thecurrent frame and PeakTop value. In an exemplary embodiment, at 892, ifthe numerator value is non-zero, then, at 894, a ratio value is assignedto the nearest integer to the expression [12 times the log base 2 of(the numerator value over the denominator)]. If the numerator value iszero, then the flow proceeds to 900. At 896, if the ratio value isbetween 0 and 25, then at 898, an intervals array keeping track ofcumulative values for a given ratio value is incremented by theexpression the square of the energies at the current frame number andcurrent PeakBottom value and the square of the energies at the currentframe number and current PeakTop value. At 900, it is determined whetherthe last PeakTop value has been processed. If not, the flow returns to890 for further processing of the next ratio based upon the newnumerator value. If so, then, the flow proceeds to 902 where it isdetermined whether the last PeakBottom value has been processed. If not,the flow returns to 884 for processing of the next ratio based upon thenew denominator value. If so, then the flow proceeds to 904 where it isdetermined whether the last frame has been processed. If not, the flowproceeds to 882 where processing of the next frame according to steps882 to 902. If the last frame has been processed, then the flow proceedsto 906 wherein the means of the interval array values are calculated. At908, the interval array or vector is normalized and the flow completesat 910.

Then, operation of the classification chain 814 may be performed.Operation of the classification chain 814 may include two phases:classification and operation. Human experts, as described above, mayundertake the classification phase. These experts assign each entry inthe data set, to one or more musical consonance classes, correspondingto some relevant perceptual consonance properties of the data. Theclassified data is then used to construct the initial classificationchain. With the initial classification chain, the classification may be“tweaked,” for example, in accordance with the process illustrated inFIG. 6C. Once the classification chain is ready for operation, it may beused to classify or otherwise relate songs according to their consonanceproperties.

Method and Process for Melodic Movement Classification

The present invention also provides a system and methods forautomatically quantifying and classifying melodic movement in a mediaentity.

As illustrated in FIG. 9A, the automatic quantification andclassification of melodic movement of a media entity may include a peakdetection and interpolation phase 905, a critical band masking phase910, a peak continuation phase 915, a melodic movement vectorcalculation 920 and 925, a principal component analysis (PCA) transformstage 930 and a classification phase 935.

Sound, such as PCM audio data, after taking the FFT at 900 is read intothe peak detection and interpolation stage 905 where the most prominentpeaks along with their energies are detected and recorded into outputmatrices. These matrices are then fed through the critical band maskingstage 910, the peak continuation stage 915, and the melodic vectorcalculation stage 920. The Melodic vector of 925 is then optionallymultiplied by the principal component matrix at 930, and fed into theclassification chain at 935, which finally determines the melodicmovement value for the sound.

With the peak detection and interpolation phase 905, peak detection isperformed on a frame-by-frame basis of an audio file by recording themaximum peak locations and energies in the frequency spectrum for eachframe. The top P, a user specified parameter, number of peaks areselected for each frame and recorded with their respective energy intooutput vectors for each frame. These peak energies and locations arethen Nth-order interpolated to increase precision. The final output istwo P×F matrices: one holding the P maximum peak locations (frequency inHz) for each frame (F total frames), and the other holding therespective energy value for the peak location in each frame. Anexemplary implementation of the peak continuation phase 915 may be thepeak continuation phase as described above in connection with FIG. 8B.

The critical band masking stage 910 filters and removes any peaks thatare masked by surrounding peaks with more energy. The amount of maskingis scalable, and this feature may be turned off completely and thus isoptional.

The peak continuation phase 915 is performed by keeping track of peaksthat persist for more than a specified number of frames, and by fillingin peaks where they may have been missed. This is accomplished by usingpeak “guides” that are initially set to the peak locations in the firstframe, and then on a frame-by-frame basis, finding suitable guides foreach peak within the frame. If a guide is found, then the peak energydata is recorded and continued, otherwise the data is discarded. Anexemplary implementation of the peak continuation phase 915 may be thepeak continuation phase as described above in connection with FIG. 8C.

The melodic vector calculation stage 920 creates a 1×24 output vectorcontaining the standard deviation of the pitch-class-movement vectorsfor each frame. This ‘pitch-class-movement’ vector is created by binningand summing the energy in the first 24 pitch classes (two octaves) foreach peak in a frame. After all frames have been calculated, theapproximate first derivative is taken with respect to time, and finallythe standard deviation to give the 1×24 melodic vector for the entiresound.

The melodic vector calculation phase 920 may be described in more detailwith reference to FIG. 9B illustrating a flow diagram of some exemplarynon-limiting pseudocode which one of ordinary skill in the art mightimplement among many different software or firmware realizations of thepresent invention. At 940, a FrameNum variable is set to loop from 1 tothe number of frames. At 945, a PeakNum variable is set to loop from 1to the number of peaks in the frame. At 950, a numerator value is set tothe peak location matrix value at the current frame and PeakNum value.At 955, if the numerator value is between or equal to either of k (aconstant) times the minimum frequency and k times the maximum frequency,then at 960, the PitchClass is determined according to a mathematicalexpression, such as: round (24 times the log base 2 of (the numeratorvalue divided by k times the minimum frequency)) minus floor (the logbase 2 of (the numerator value divided by k times the minimumfrequency)). If the numerator value is outside the boundaries of k timesthe minimum frequency and k times the maximum frequency, then the flowproceeds to 975. At 965, if the PitchClass is between 0 and 25, then at970, the Melodic matrix value for that FrameNum and PitchClasscumulatively receives the value of the Energies matrix for that FrameNumand PeakNum. If the PitchClass is not between 0 and 25, then the flowproceeds to 975. At 975 and 980, either the PeakNum or FrameNum loopvalues are updated until 950 to 970 have been performed for each of theframes and each of the peaks. At 985, a first order difference vectormatrix is formulated from the melodic vector matrix. At 990, thestandard deviations of the first order difference vector matrix valuesare calculated and the flow ends at 995.

With the optional principal component transform phase 930, the melodicvector is concatenated and the matrix is multiplied by a principalcomponent matrix. This transforms the vector into a principal componentcoordinate system defined by the classification chain in order toextract the salient features of the information represented thereby.

The operation of the classification chain 935 may be broken down into aclassification phase and an operation phase. As described in more detailabove, human experts undertake the classification phase. These expertsassign each entry in the data set to one or more classes correspondingto the relevant perceptual properties of the melodic movement of thedata. The classified data is then used to construct the initialclassification chain for classification of media entities according totheir melodic movement properties. As mentioned, principal componentanalysis may be used to reduce the amount of data, and to removeredundancy in the chain.

System and Method for Tempo Classification

One application for the above-described classification chain is withrespect to a measurement of perceived sound tempo. Tempo is the “pulse”or the “heart beat” of a musical performance. In essence, the tempoconveys the perceived velocity of the performance. Tempo may bedescribed as the rate of motion or activity, or as the rate of speed ofa musical piece or passage indicated by one of a series of directionse.g., largo, presto, or allegro. In one aspect, the present inventionthus provides a system and method of determining tempo given a datarepresentation of a musical performance. This may be used in anautomated system, for example, to classify a large database of musicaccording to its tempo properties, as described above in connection withthe adaptive media processes of the invention. It can be usedindependently to give a reliable tempo determination of a given piece ofmusic.

The input to the tempo classification system is a media entity, such asa song. A media entity, such as a song, may be represented in variety ofdigital formats, whether or not converted from analog. Such formatsinclude a computer data file, such as a “.wav” file extracted from amusic compact disc or an “.mp3.” Using the tempo classification methodsof the invention, as described below, this data is distilled to a morecompact representation that is suited to addition to a classificationchain. The output of the classification chain, after training of theclassification chain, thus provides a reliable estimate of the tempo.

The data used is generally in the form of monophonic “raw” digital data,such as PCM data. To form such raw data, various components of a datastream may require stripping. For example, a track ripped from a CD maybe stripped of any header information and converted to raw mono 44 kHz16 bit data. An “mp3” may be converted to a “wav” file and converted tomono, along with removing any header information. Any format, however,may be processed to provide for uniform data representation. Thus, thepresent invention can also work with data of other sampling rates andresolutions provided the audio quality is not noticeably compromised.

In accordance with the tempo measurement of the invention, for a 44 kHzsampling rate, the data is decimated to a representative envelope 1024times smaller than its original size. For other sampling rates thedecimation factor is adjusted to yield approximately a 43 Hz samplingrate. This sampling rate in conjunction with the rest of the processing,while not a non-limiting design choice, provides an ideal resolution fortempo determination. In the case of 44 kHz sampled media entity, thedecimation may be performed by taking the square root of the sum of thesquares of the 1024 samples, although other well known sub-sampling oraveraging techniques may be used.

In an exemplary implementation, the method for determining tempoproperties of a media entity exaggerates and accentuates the tempocharacteristics of an envelope generated by processing the raw data. Atthe same time, the method also smoothes the envelope and removes fromthe envelope biases and trends. This includes performing a first orderdifference calculation followed by performing half wave rectification. Amedian filter may be applied to smooth out aberrations, biases and/ortrends. Then, after the mean value is subtracted, the data may be halfwave rectified again. Then, another first order difference calculationmay be performed, followed again by half wave rectification.

The resulting waveform from the above processing is used to generate thefinal data that may be input, for example, to a classification chain.The final data consists of 130 values or “bins” reflective of thedifferent correlation strengths at different time periods orfrequencies. The final data is generated by looping through thedecimated waveform and accumulating the base two logarithm of theproduct of pairs of points located at a fixed offset from each other.This is done 130 times for offsets spanning from 1 to 130 samples.

The 130 values of the final data are fed into a classification chainthat is built upon an existing database classified by humans. Then, theoverall distance between the input data vector and each individual pointin the database is computed. First, the distances between each of the130 individual dimensions of the input vector versus each individualpoint in the database are measured. The distances for each dimension aresquared and added together. The square root of the sum of these valuesgives the overall distance between the input vector and each individualpoint in the database. If this value is below a given threshold for agiven input vector and database point pair, this distance value, alongwith the tempo value associated with the specific point from thedatabase are added to a table.

After cycling through the entire list of points in the database, a tableof distance and tempo values is generated. The distance values aretranslated into confidence values which are proportional to1/distance{circumflex over ( )}4. The tempo values are translated into aclass and octave component by dividing by 10 and taking the base 2logarithm of the result. The integer portion represents the octavecomponent and the fractional part represents the class component.

The tempo components are then averaged in a manner to find the besttempo representation for the input vector. First, each class componentvalue in the table is mapped to an angular representation by multiplyingby 2π. The sine and cosine of the resulting values are separately takenand multiplied by the corresponding confidence value. All of thegenerated sine values are accumulated together and all of the generatedcosine values are accumulated together. The sum of the sine values isdivided by the sum of the cosine values and a four quadrant inversetangent is taken. The resulting angle, ranging between 0 and 2π ismapped back to a value between zero and one, determining the overallclass component estimation for the input vector.

The class component estimation is used to determine a threshold to helpgenerate the overall octave component estimation of the input vector.Each class entry in the table has an associated octave component. If theoverall class component estimation is greater than 0.5, each class entryis compared to this value minus 0.5, and if it is less, thecorresponding octave component is decremented. Inversely, if the overallclass component estimation is less than 0.5, each tempo entry iscompared to this value plus 0.5, and if it is more, the correspondingoctave component is incremented.

The octave components may be used as indexes into an accumulative array.The array is initialized to zeros and for each table entry, and theoctave component determines the index to which the correspondingconfidence value is accumulated. By cycling through all of the tableindices, the table index with the largest value is assigned to theoverall octave component estimation for the input vector.

This process returns two values. The first value is the final tempoestimation. In an exemplary calculation, this value is obtained bycalculating the result of the expression: 2{circumflex over ( )}(overalltempo component estimation plus overall harmonic component estimation)multiplied by 10. The second value returned is the overall confidence.In an exemplary calculation, this value is obtained by calculating theresult of the expression: the square root of the sum of the square ofthe accumulated sine values and the square of the accumulated cosinevalues.

The tempo calculations may be described in more detail with reference toFIG. 10 illustrating an exemplary non-limiting flow diagram that one ofordinary skill in the art might implement in choosing among manydifferent realizations of tempo calculation in accordance with thepresent invention. At 1020, audio data is received in 16 bit 44 kHz monoPCM format. Various other formats may be accommodated as well. At 1030,the audio data is decimated, sub-sampled and/or averaged to a 43 Hzsampling rate by collapsing 1024 samples into 1 sample. As mentioned,this may be done by taking the square root of the sum of the squares ofblocks of 1024 samples. At 1040, the data is processed to generate anenvelope of data that accentuates tempo characteristics whilesuppressing biases and/or trends. This may be done by taking the firstorder difference, then half wave rectifying, then applying a medianfilter, then subtracting the mean value, again half wave rectifying,then taking the first order difference and once again half waverectifying. At 1050, 130 bins of data are generated, with each bin beingdefined as the “correlation strength” for the period defined by the binnumber times 1/43. For example, bin number 43 corresponds to a period of1 second or 60 beats per minute. The calculation is similar to anautocorrelation function, except, inter alia, that the log base 2 of theproduct of the data is accumulated.

At 1060, the data may be fed into the classification chain. Then, forevery calculated distance that is below a defined threshold, thecorresponding classified tempo along with the distance is added to atable. In an exemplary implementation, the distance values in the tableare translated into confidence values by taking 1/(distance{circumflexover ( )}4). Each tempo value is divided by ten and the log base 2 ofthe result is taken. The integer portion is the “harmonic component”entry and the fractional portion is the “tempo component” entry. At1070, the tempo components are translated into vector representation.The range from 0 to 1 is mapped to the range from 0 to 2π. The sine andcosine of the angles are taken and multiplied by the correspondingconfidence value. These sine and cosine components for the entire tablemay be accumulated separately to create an overall representativevector. The angle of this vector is mapped back to a range from 0 to 1to give an overall confidence of the tempo classification. At 1080, theharmonic component of each table entry is evaluated. If thecorresponding tempo component meets one of the following criteria, theharmonic component is modified. If the tempo entry is less than theoverall tempo classification minus 0.5, the harmonic component entry isdecremented. If the tempo entry is greater than the overall tempoclassification plus 0.5, the harmonic component is incremented. Eachharmonic component table entry “votes” for its value with a weightproportional to the corresponding confidence value. The most prominentvalue is used as the overall harmonic component. At 1090, the overalltempo is calculated by evaluating the expression: 10 times 2{circumflexover ( )}(overall harmonic component plus overall tempo component).

In addition to the advantage of merging perceptual or human classifiedtempo properties with the DSP tempo properties of media entities, theabove-described methods of tempo classification are significantly fasterthan the techniques utilized in the prior art. Using a classificationchain built from at least 100,000 songs, for example, the accuracy iscomparable or better. The method also returns a confidence factor, whichflags input data that cannot be classified reliably. The sameclassification chain can simultaneously be used to determine otherparameters, such as time signature and any other property describedabove. Other advantages inherent in the use of a classification chain inaccordance with the present invention are described in more detailabove.

The media entities contemplated by the present invention in all of itsvarious embodiments are not limited to music or songs, but rather theinvention applies to any media to which a classification technique maybe applied that merges perceptual (human) analysis with acoustic (DSP)analysis for increased accuracy in classification and matching. Whilevarious embodiments of the present invention have been described inconnection with sonic, consonance, tempo, melodic movement properties ofmedia entities, it is to be understood that any combination orpermutation thereof is considered when classifying a media entity for aset of properties for a classification chain, and that additionalproperties may be adapted to the classification chain as well. Forexample, by performing automatic DSP processing of a media entity for aproperty to be adapted, wherein human experts have previously classifiedthe corresponding perceptual characteristic(s) suited to the property,and then forming a vector for inclusion in the classification chain, aninitial classification for the adapted property may be formed. Then, asnew unclassified media entities are included in the system, theclassification chain can be “tweaked,” as described above, to improvethe number of successful responses when classifying new, unclassifiedmusic for that adapted property.

The various techniques described herein may be implemented with hardwareor software or, where appropriate, with a combination of both. Thus, themethods and apparatus of the present invention, or certain aspects orportions thereof, may take the form of program code (i.e., instructions)embodied in tangible media, such as floppy diskettes, CD-ROMs, harddrives, or any other machine-readable storage medium, wherein, when theprogram code is loaded into and executed by a machine, such as acomputer, the machine becomes an apparatus for practicing the invention.In the case of program code execution on programmable computers, thecomputer will generally include a processor, a storage medium readableby the processor (including volatile and non-volatile memory and/orstorage elements), at least one input device, and at least one outputdevice. One or more programs are preferably implemented in a high levelprocedural or object oriented programming language to communicate with acomputer system. However, the program(s) can be implemented in assemblyor machine language, if desired. In any case, the language may be acompiled or interpreted language, and combined with hardwareimplementations.

The methods and apparatus of the present invention may also be embodiedin the form of program code that is transmitted over some transmissionmedium, such as over electrical wiring or cabling, through fiber optics,or via any other form of transmission, wherein, when the program code isreceived and loaded into and executed by a machine, such as an EPROM, agate array, a programmable logic device (PLD), a client computer, avideo recorder or the like, the machine becomes an apparatus forpracticing the invention. When implemented on a general-purposeprocessor, the program code combines with the processor to provide aunique apparatus that operates to perform the indexing functionality ofthe present invention. For example, the storage techniques used inconnection with the present invention may invariably be a combination ofhardware and software.

While the present invention has been described in connection with thepreferred embodiments of the various figures, it is to be understoodthat other similar embodiments may be used or modifications andadditions may be made to the described embodiment for performing thesame function of the present invention without deviating therefrom. Forexample, while exemplary embodiments of the invention are described inthe context of music data, one skilled in the art will recognize thatthe present invention is not limited to the music, and that the methodsof tailoring media to a user, as described in the present applicationmay apply to any computing device or environment, such as a gamingconsole, handheld computer, portable computer, etc., whether wired orwireless, and may be applied to any number of such computing devicesconnected via a communications network, and interacting across thenetwork. Furthermore, it should be emphasized that a variety of computerplatforms, including handheld device operating systems and otherapplication specific operating systems are contemplated, especially asthe number of wireless networked devices continues to proliferate.Therefore, the present invention should not be limited to any singleembodiment, but rather construed in breadth and scope in accordance withthe appended claims.

1. A method for determining tempo in an input piece of music,comprising: creating a compact representation of the input piece ofmusic; processing of the compact representation in a way thataccentuates its tempo characteristics; generating bins of data from theprocessed compact representation, reflective of the differentcorrelation strengths at different time periods or frequencies; andfeeding the bins of data into a classification chain.
 2. The method ofclaim 1, wherein the compact representation of the input piece of musicis created by taking the square root of the sum of the squares of blocksof 1024 samples.
 3. The method of claim 1, wherein the processing of thecompact representation is done by first taking of a first orderdifference, a first half wave rectifying, applying median filter,subtracting a mean value, a second half wave rectifying, second takingof a first order difference, and a third half wave rectifying.
 4. Themethod of claim 1, wherein 130 bins of data are generated, each binbeing the correlation strength of the period defined by the bin numbertime 1/43.
 5. The method of claim 1, wherein the classification chain isbuilt upon an existing database classified by humans.